Skip to content

feat: add callback support to aws batch executor#62984

Open
dondaum wants to merge 1 commit into
apache:mainfrom
dondaum:feat/batch-executor-callback-support
Open

feat: add callback support to aws batch executor#62984
dondaum wants to merge 1 commit into
apache:mainfrom
dondaum:feat/batch-executor-callback-support

Conversation

@dondaum
Copy link
Copy Markdown
Contributor

@dondaum dondaum commented Mar 6, 2026 •
edited
Loading

Implements executor callback support for the AWS BatchExecutor.

related: #62887

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@dondaum dondaum requested a review from o-nikolas as a code owner March 6, 2026 09:32
@boring-cyborg boring-cyborg Bot added area:providers provider:amazon AWS/Amazon - related issues labels Mar 6, 2026
@ferruzzi ferruzzi mentioned this pull request Mar 10, 2026
Open
2 tasks
o-nikolas
o-nikolas reviewed Mar 11, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@o-nikolas o-nikolas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still see a lot of other methods with names, types, variables, comments all mentioning tasks where it should be workloads that need cleaning up.

@ferruzzi should have a look too of course

Comment thread providers/amazon/src/airflow/providers/amazon/aws/executors/batch/batch_executor.py Outdated
Comment thread providers/amazon/src/airflow/providers/amazon/aws/executors/batch/batch_executor.py Outdated
Comment thread providers/amazon/src/airflow/providers/amazon/aws/executors/batch/batch_executor.py
@potiuk
Copy link
Copy Markdown
Member

potiuk commented Mar 12, 2026 •
edited
Loading

@dondaum A few things need addressing before review — see our Pull Request quality criteria.

  • ⚠️ Unresolved review comments: This PR has 3 unresolved review threads from maintainers. Please review and resolve all inline review comments before requesting another review. You can resolve a conversation by clicking 'Resolve conversation' on each thread after addressing the feedback. See pull request guidelines.

Note: Your branch is 278 commits behind main. Please rebase and push again to get up-to-date CI results.

No rush.

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

@dondaum dondaum force-pushed the feat/batch-executor-callback-support branch from 571de3f to c591fbe Compare March 12, 2026 10:08
@dondaum
Copy link
Copy Markdown
Contributor Author

dondaum commented Mar 12, 2026

I still see a lot of other methods with names, types, variables, comments all mentioning tasks where it should be workloads that need cleaning up.

@ferruzzi should have a look too of course

I changed the reference from task to workload where I could make a change. There is a method try_adopt_task_instances() which I have not changed since it is part of the base class. But technically it should work on both tasks and callbacks.

@dondaum dondaum force-pushed the feat/batch-executor-callback-support branch 2 times, most recently from 67332d4 to 00c71ae Compare March 12, 2026 11:11
@eladkal eladkal requested review from o-nikolas and vincbeck March 12, 2026 18:04
shivaam
shivaam reviewed Mar 15, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@shivaam shivaam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Left a comment.

@ferruzzi ferruzzi mentioned this pull request Mar 16, 2026
Draft
1 task
@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Apr 2, 2026
@shivaam shivaam mentioned this pull request Apr 15, 2026
Open
1 task
@dondaum dondaum force-pushed the feat/batch-executor-callback-support branch 2 times, most recently from 639de36 to a651768 Compare April 20, 2026 12:34
ferruzzi
ferruzzi reviewed Apr 20, 2026
View reviewed changes
Comment thread providers/amazon/src/airflow/providers/amazon/aws/executors/batch/batch_executor.py Outdated
Comment thread providers/amazon/src/airflow/providers/amazon/aws/executors/batch/batch_executor.py Outdated
Comment thread providers/amazon/src/airflow/providers/amazon/aws/executors/batch/batch_executor.py Outdated
Comment thread providers/amazon/src/airflow/providers/amazon/aws/executors/batch/batch_executor.py
Comment thread providers/amazon/src/airflow/providers/amazon/aws/executors/batch/batch_executor.py Outdated
Comment thread providers/amazon/src/airflow/providers/amazon/aws/executors/batch/utils.py Outdated
Comment thread providers/amazon/tests/unit/amazon/aws/executors/batch/test_batch_executor.py Outdated
@dondaum dondaum force-pushed the feat/batch-executor-callback-support branch 2 times, most recently from f5a8db8 to 3867506 Compare April 24, 2026 07:51
@ferruzzi ferruzzi mentioned this pull request Apr 24, 2026
Closed
@ferruzzi
Copy link
Copy Markdown
Contributor

ferruzzi commented May 12, 2026

@dondaum - Resolve that merge conflict and I'll approve once the CI is green 👍

@dondaum dondaum force-pushed the feat/batch-executor-callback-support branch from 3867506 to 0f3e99f Compare May 12, 2026 07:42
@dondaum
Copy link
Copy Markdown
Contributor Author

dondaum commented May 12, 2026

@dondaum - Resolve that merge conflict and I'll approve once the CI is green 👍

done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ferruzzi ferruzzi ferruzzi approved these changes

@o-nikolas o-nikolas o-nikolas approved these changes

@vincbeck vincbeck Awaiting requested review from vincbeck

@hussein-awala hussein-awala Awaiting requested review from hussein-awala hussein-awala is a code owner

@dheerajturaga dheerajturaga Awaiting requested review from dheerajturaga dheerajturaga is a code owner

@XD-DENG XD-DENG Awaiting requested review from XD-DENG XD-DENG is a code owner

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@pierrejeambrun pierrejeambrun Awaiting requested review from pierrejeambrun pierrejeambrun is a code owner

+1 more reviewer

@shivaam shivaam shivaam left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area:providers provider:amazon AWS/Amazon - related issues ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@dondaum @potiuk @ferruzzi @shivaam @o-nikolas